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1.  Introduction 


The  Augmented  REality  Sandtable  (ARES)  is  a  research  and  development  testbed 
with  the  aim  to  improve  battlespace  visualization  in  order  to  provide  a  user-defined 
common  operating  picture  at  the  point  of  need.  As  a  physical  platform,  ARES 
consists  of  several  modalities,  including  but  not  limited  to  the  following: 

•  A  traditional  sand  table  filled  with  play  sand  and  supplemented  with 
commercial  off-the-shelf  components  that  allow  the  user  to  view  and 
interact  with  visual  representations  of  an  area  of  operations  (AO)  and  related 
data. 

•  A  mobile  software  application  that  provides  an  overview  of  the  AO  with 
geospatial  terrain  data  and  additional  layers  of  data  for  interaction  and 
analysis. 

•  An  application  that  displays  terrain  and  other  data  in  the  AO  via  mixed- 
reality  headsets  (e.g.,  Microsoft  HoloLens  or  HTC  Vive). 

In  addition  to  the  various  modalities,  government-owned  ARES  software  provides 
geospatial  terrain  information  and  map  images,  tracks  the  sand  topography,  and 
allows  users  to  build  or  edit  tactical  mission  plans.  The  software  serves  the  data  to 
client  applications  that  then  provide  the  data  to  users  via  one  of  the  modalities. 
Figure  1  shows  how  the  modalities  work  together  in  one  ecosystem. 
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Fig.  1  ARES  platform  concept  showing  multiple  modalities 
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As  a  testbed  for  research,  ARES  is  primarily  concerned  with  human  factors  research 
in  the  areas  of  information  visualization,  multimodal  interaction,  and  human 
performance  assessment.  The  purpose  of  this  technical  note  is  to  discuss  completed, 
ongoing,  and  planned  research  and  provide  an  overall  strategy  for  future  research. 
In  this  context,  “research”  refers  to  activities  conducted  to  gain  insight  into  basic  or 
applied  areas  of  inquiry  related  to  ARES,  as  differentiated  from  the  “research  and 
development”  required  to  expand  the  capabilities,  supported  analyses,  or  features 
of  the  ARES  platform. 

A  US  Army  Research  Laboratory  (ARL)  special  report  was  prepared  in  2015  that 
includes  a  description  of  the  ARES  platform,  relevance  to  the  Army,  similar  efforts 
to  date,  existing  and  planned  future  capabilities,  and  discussion  of  many  key  leader 
engagements  (Amburn  et  al.  2015).  While  the  2015  ARES  report  included  passing 
mention  of  research  and  evaluation  activities,  the  present  technical  note  instead 
delves  deeper  into  the  research  activities.  This  document  should  not  be  viewed  as  a 
comprehensive  agenda  of  work  to  be  completed  but  is  instead  analogous  to  the  work 
of  an  urban  planner.  In  that  domain,  deliverables  might  include  conceptual 
drawings  showing  generic  blocks  with  intended  buildings  or  usages,  but  actual 
construction  might  substantially  differ  both  in  form  and  purpose.  Similarly,  our 
plan  is  intended  to  summarize  the  work  performed  to  date,  project  the  regions 
within  which  research  will  be  developed,  and  establish  the  predominant  and 
supplementary  research  questions  that  support  future  activities.  However,  this  plan 
is  subject  to  change  with  changing  requirements,  new  technological  developments, 
or  interest  and/or  funding  for  research  with  new  use  cases. 

2.  Research  Questions  and  Areas  for  ARES  Research 


Fundamentally,  the  predominant  research  question  underlying  all  ARES  research 
activities  is  the  following:  What  improvements  in  battlespace  visualization  and 
decision-making  aid  in  providing  a  common  operating  picture  at  the  point  of  need 
and  best  meet  user  requirements?  The  ARES  platform  brings  together  development 
in  several  areas  to  provide  improvements  to  traditional  means  of  battlespace 
visualization,  including:  interactive  displays;  tangible  user  interfaces;  and 
augmented,  virtual,  and  mixed  reality.  Each  of  these  topics  contributes  to  one  of  the 
research  areas  under  investigation  to  provide  advancements  that  help  answer  the 
predominant  research  question  above.  These  advancements  may  take  the  form  of 
improvements  to  the  ARES  interface  and  interaction  among  modalities, 
development  of  other  interface  device(s),  or  generalizable  research  that  answers 
important  questions  of  interest,  ultimately  yielding  new  tools  for  Soldiers  that 
contribute  to  Army  readiness.  This  section  addresses  the  predominant  research 
question,  lanes  of  research,  and  an  overall  strategy  for  achieving  research  goals. 
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2.1  ARES  Motivation  and  Mission 


An  augmented  reality  sand  table  has  the  potential  to  enhance  spatial  awareness, 
improve  visualization  of  the  battlespace,  increase  engagement,  enable  distributed 
collaboration,  and  save  time  when  authoring  3-D  terrains  and  scenarios  (Amburn 
et  al.  2015).  Research  for  the  ARES  platform  supports  many  of  the  following  Army 
Warfighting  Challenges  (ACIC  2017): 

•  AWFC  #1:  Develop  Situational  Understanding 

•  AWFC  #5:  Counter  Weapons  of  Mass  Destruction 

•  AWFC  #6:  Conduct  Homeland  Operations 

•  AWFC  #8:  Enhance  Realistic  Training 

•  AWFC  #9:  Improve  Soldier,  Feader,  and  Team  Performance 

.  AWFC  #14:  Ensure  Interoperability  and  Operate  in  Joint,  Inter- 
organizational,  Multi-national  (JIM)  Environment 

•  AWFC  #15:  Conduct  Joint  Combined  Arms  Maneuver 

•  AWFC  #17/18:  Employ  Cross-Domain  Fires 

•  AWFC  #19:  Exercise  Mission  Command 

Research  for  the  ARES  platform  may  also  support  the  following  ARE  Key 
Campaign  Initiatives  (KCIs),  as  identified  in  ARF’s  Technical  Implementation 
Plan  2016-2020  (ARE  2016): 

Information  Sciences  Campaign 

•  KCI-IS-2:  Taming  the  Flash-Floods  of  Networked  Battlefield  Information 

•  KCI-IS-3:  Acting  Intelligently  in  a  Dynamic  Battlefield  of  Information, 
Agents,  and  Humans 

•  KCI-IS-4:  Sensing  and  Information  Fusion  for  Advanced  Indications  and 
Warnings 

Human  Sciences  Campaign 

•  KCI-HS-1:  Robust  Human  and  Machine  Hybridization 

•  KCI-HS-2:  Multi-faceted  Assessment  of  Soldier  Variability 

•  KCI-HS-3:  Training  Effectiveness  Research 
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ARES  is  certainly  not  the  first  augmented  reality  sand  table;  the  2015  ARES  report 
tabulated  25  efforts  related  to  visualizing  spatial  data  on  virtual  or  sand  table 
interfaces  stretching  back  20  years.  The  overwhelmingly  positive  reaction  to  ARES 
and  the  existence  of  various  other  virtual/augmented  reality  sand  table  research  and 
development  efforts  bear  testament  to  the  great  interest  that  exists  in  improving 
traditional  means  of  visualizing  and  interacting  with  geospatial  data  relevant  to  the 
battlespace. 

It  is  also  important  to  emphasize  that  the  ARES  research  program  is  not  limited  to 
specific  modalities  (e.g.,  sand  table).  For  instance,  research  might  consider  whether 
an  augmented  reality  projection  on  a  wall  with  an  interactive  natural  user  interface 
enables  new  or  different  visualization,  analysis,  and  decision-making  capabilities. 
To  address  this  consideration — and  many  others  in  this  domain — it  is  helpful  to 
consider  each  part  of  the  predominant  research  question  for  the  ARES  program. 
The  following  subsections  address  the  3  constituent  components:  improved 
battlespace  visualization  and  decision  making,  common  operating  picture  defined 
by  user  requirements,  and  point  of  need. 

2.1.1  Improved  Battlespace  Visualization  and  Decision  Making 

Advancements  in  geospatial  terrain  visualization  offer  direct  application  to  the 
ARES  platform.  Many  Geospatial  Information  System  software  suites  offer  the 
ability  to  import  terrain  information  and  build  graphical  layers  on  top  of  the  base 
layer.  However,  these  computer-based  applications  do  not  inherently  offer  a  true 
3-D  representation  of  the  data  as  they  are  depicted  on  a  2-D  computer  monitor. 
Novel  systems  have  begun  to  bring  this  data  into  3-D  space  and  combine  it  with 
intuitive  user  interfaces.  For  instance,  Mitasova  et  al.  (2012)  discuss  various 
techniques  for  constructing  interactive  3-D  multisurface  visualizations  with 
application  to  tangible  environments;  Fuhrmann  et  al.  (2009)  investigated  the  use 
of  geospatial  holograms  for  wayfinding  and  route  planning  for  use  by  Special 
Weapons  and  Tactics  (SWAT)  teams.  Providing  additional  layers  of  information 
and  analyses  on  top  of  the  geospatial  data  enhances  the  ability  of  decision  makers 
to  make  more  informed  decisions.  For  instance,  additional  visualization  capabilities 
might  enable  better  perception  of  the  terrain  and  dynamic  presence  of  units  and  the 
employed  tactics.  It  is  essential  to  conduct  research  on  the  human  factors  of  such 
improvements  to  investigate  their  effect  on  the  performance  of  decision  makers  and 
optimize  the  presentation. 
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2.1.2  Common  Operating  Picture  Defined  by  User  Requirements 

A  key  characteristic  of  the  ARES  platform  is  the  customizability  and  interactivity 
of  the  geospatial  visualization.  In  other  words,  the  platform  enables  decision  makers 
to  select  the  information  that  is  relevant  to  them  to  create  a  picture  of  the  battlespace 
that  supports  decisions  they  have  to  make.  Other  literature  refers  to  this  as  a  User- 
Defined  Common  Operating  Picture  (UDOP).  Mulgund  and  Landsman  (2007) 
describe  the  UDOP  operational  concept  and  a  prototype  implementation;  Hiniker 
et  al.  (2007)  discuss  plans  to  assess  an  interface  designed  with  UDOP  principles. 
Research  should  seek  to  better  understand  the  types  of  information  relevant  to 
certain  classes  of  users,  mechanisms  for  customizing  the  interface,  and  how  best  to 
integrate  the  various  data  to  present  an  enhanced  common  operating  picture. 

2.1.3  Point  of  Need 

The  point  of  need  refers  to  the  venue — both  environment  and  setting — where 
decisions  must  be  made  or  training  needs  to  be  conducted.  An  ideal  decision 
support  tool  easily  adapts  to  varying  points  of  need,  whether  in  a  classroom  or 
operational  setting,  or  in  a  conference  room  or  outdoor  environment.  The  ARES 
sand  table  interface  extends  to  both  classroom  and  operational  settings  but  is  limited 
to  controlled  indoor  environments;  the  mobile  and  mixed  reality  interfaces  offer 
greater  portability  but  do  not  have  a  tangible  component.  Areas  of  research  to 
consider  are  how  the  platform  can  be  extended  to  other  environments,  how  various 
modalities  may  interact  to  better  address  adaptability/scalability  to  the  point  of 
need,  and  what  the  implications  are  for  the  overall  value  to  the  decision-maker. 

The  following  sections  discuss  each  of  the  research  areas  that  characterize  ARES 
research  activities. 

2.2  Information  Visualization  Research  Area 

The  critical  question  in  information  visualization  is  how  best  to  transform  data  into 
something  that  people  can  understand  for  optimal  decision-making  (Ware  2012). 
ARES  is  fundamentally  a  system  for  visualizing  a  battlespace  and  providing  tools 
that  enable  better  decisions  to  be  made.  The  information  visualization  research  area 
investigates  questions  in  this  domain.  In  a  military  context,  information 
visualization  has  been  described  as  the  cohesion  of  information  characteristics  and 
human  cognitive  processes  that  are  embodied  and  situated  by  2  requirements.  The 
first  is  battle  command,  which  entails  decision-making  and  visualizing  the  current 
and  future  state  of  operations.  The  second  is  situational  awareness,  which  entails 
the  ability  to  identify,  process,  and  comprehend  the  critical  elements  of  information 
about  what  is  happening  (Ntuen  2009). 
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Information  visualization  elevates  the  comprehension  of  information  by  fostering 
rapid  correlation  and  perceived  associations  (Livnat  et  al.  2005).  The  design  of  the 
information  visualization  platform  must  support  the  decision-making  process, 
identifying  and  characterizing  problems,  and  determining  appropriate  responses.  In 
the  context  of  research  for  the  ARES  platform,  the  area  of  greatest  interest  is  the 
nexus  of  mixed  reality  and  data  visualization  to  ensure  that  optimization  of 
information  visualization  techniques  for  battle  command  and  situational  awareness 
are  incorporated.  This  inherently  involves  large  datasets  (e.g.,  geospatially 
distributed  model  outputs). 

Research  conducted  in  this  area  has  explored  information  visualization  in  specific 
contexts  (e.g.,  battlespace  visualization),  the  use  of  multiple  views  to  visually 
convey  information  (Baldonado  et  al.  2000),  and  situational  awareness  for  decision¬ 
making.  In  addition,  other  research  has  attempted  to  support  information 
visualization  research  and  development  by  creating  a  taxonomy  to  codify  human 
perceptual  and  cognitive  capabilities  and  limitations,  independent  of  domain, 
thereby  providing  a  means  to  empirically  assess  and  compare  research  outcomes 
(Pfitzner  et  al.  2001).  Colleagues  at  ARL  are  also  conducting  research  in 
information  visualization  that  may  be  relevant  to  the  ARES  platform  (e.g.,  Chen 
2005;  Chen  et  al.  2014;  Hansberger  2015);  any  future  efforts  in  similar  topics 
should  first  consult  these  or  other  ARL  researchers  to  gain  insight  on  lessons 
learned  or  points  for  collaboration. 

Research  questions  of  interest  may  include  the  following: 

•  Can  a  defined  taxonomy  help  delineate  various  factors  in  battlespace 
visualization  that  will  serve  to  assist  in  evaluating  and  assessing  the 
effectiveness  of  ARES? 

•  How  would  using  various  information  visualization  techniques  in  a 
distributed  environment  affect  team  performance? 

•  How  do  various  information  visualization  implementations  affect 
performance  on  tasks  related  to  battlespace  visualization  and  what  types  of 
tasks  or  scenarios  are  best  suited  to  the  various  implementations? 

•  How  might  various  information  visualization  techniques  help  users 
visualize  interactions  with  intelligent  agents  in  the  battlespace  and  improve 
human-agent  teaming? 

•  Do  users  benefit  from  controlling  the  amount  and  type  of  information 
presented  to  them  on  ARES? 
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Do  various  components  of  the  ARES  platform  enhance  users’  situational 
awareness,  and  to  what  extent? 


•  What  are  the  limits  of  visual  perception  that  are  necessary  to  allow  users  to 
perceive  terrain  features  (e.g.,  should  a  hill  be  3  inches  or  12  inches  to 
provide  maximum  situational  awareness)? 

•  Do  the  same  information  visualization  techniques  apply  across  the  ARES 
modalities  (i.e.,  sand  table,  mobile  tablet,  and  mixed  reality  headset)? 

•  Do  some  users  learn  better  with  using  a  real  3-D  display  than  with  virtual 
3-D  or  2-D/printed  maps? 

Information  visualization  research  should  be  formulated  in  the  context  of  how  it 
relates  to  battle  command  doctrine,  situational  awareness,  and  developing  a 
taxonomy  in  which  to  work. 

2.3  Multimodal  Interaction  Research  Area 

A  goal  of  the  ARES  program — reflected  in  the  predominant  research  question — is 
to  provide  decision  makers  with  battlespace  visualization  tools  at  the  point  of  need. 
ARES  accomplishes  this  via  the  use  of  virtual,  augmented,  or  mixed  reality  in 
multiple  modalities  (Fig.  1).  The  multimodal  interaction  research  area  is  concerned 
with  how  various  modalities  moderate  the  experience  and  affect  user  performance. 
Until  very  recently,  ARES  has  relied  primarily  on  a  single  visual-tactile  interface 
(i.e.,  the  sand  table);  future  research  should  consider  the  interaction  among  multiple 
interfaces  that  may  also  include  other  new  modalities  (e.g.,  gestures,  auditory  cues, 
voice  commands). 

Multimodal  interfaces  may  be  characterized  as  systems  that  respond  to  inputs  in 
more  than  one  modality  or  communication  channel — for  instance,  speech,  gesture, 
writing,  touch,  etc.  (Jaimes  and  Sebe  2007).  As  a  broad  area  of  research, 
multimodal  interaction  seeks  to  research  and  develop  technologies,  methods,  and 
interfaces  that  make  full  use  of  human  capabilities  to  interface  with  a  system.  Turk 
(2014)  provides  a  good  review  of  relevant  literature  in  this  area.  Multimodal 
integration  is  an  area  ripe  for  exploration — better  understanding  individual 
modalities,  how  and  when  to  integrate  multiple  channels  in  models,  and  exploring 
the  full  range  of  modality  combinations  are  all  research  challenges. 

Many  examples  in  the  literature  have  explored  different  modes  of  interacting  with 
information  using  nonconventional  displays  that  are  directly  applicable  to  the 
modalities  employed  by  the  ARES  platform,  including  1)  interactive  tabletops  (e.g., 
Annett  et  al.  2011;  Alnusayri  2015),  2)  tangible  user  interfaces  (e.g.,  Ishii  2008; 
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Maquil  2016),  and  3)  augmented  reality  and  holograms  (e.g.,  Fuhrmann  et  al. 
2009).  For  these  types  of  interfaces,  an  important  consideration  is  that  users 
appropriately  tailor  displays  to  the  task  at  hand  and  select  an  appropriate  level  of 
detail  for  their  interaction — factors  that  should  be  assessed  empirically  (Hegarty  et 
al.  2012).  Large-format  interfaces — for  instance,  interactive  tabletops  (Maldonado 
2014)  or  projector-based  augmented  reality  (Marner  et  al.  2014) — often  facilitate 
collaboration.  Tangible  user  interfaces  may  facilitate  inquiry  but  may  not  lead  to 
more  thorough  interaction  or  deeper  questions  about  the  content  (Ma  et  al.  2015), 
though  when  used  for  learning  may  yield  greater  learning  gain  (Schneider  et  al. 
2011).  Voice  command  and  auditory  feedback — perhaps  in  the  form  of  a  natural 
language  interface — present  another  avenue  for  interaction  in  the  ARES  ecosystem 
that  should  also  be  explored  in  future  research.  As  with  the  information 
visualization  research  area,  there  are  at  least  a  few  colleagues  at  ARL  who  are  also 
conducting  research  in  various  types  of  (and  combinations  of)  interfaces  that  may 
be  relevant  to  the  ARES  platform  (e.g.,  Elliott  et  al.  2009;  Myles  and  Kalb  2015). 
Researchers  working  on  future  efforts  in  similar  areas  should  remain  vigilant  for 
lessons  learned  or  points  of  collaboration. 

Some  sample  research  questions  of  interest  to  the  ARES  program  in  this  research 
area  include  the  following: 

•  What  are  the  benefits  and  drawbacks  to  the  user  of  various  types  of 
interfaces  (e.g.,  sand  table,  mobile  tablet,  and  mixed  reality  headsets)? 

•  Does  the  use  case  affect  user  performance  for  the  various  modalities? 

•  How  might  each  interface  and  its  mode  of  interaction  affect  users’  ability  to 
visualize  and  fuse  information  coming  from  a  multitude  of  sensors  and  data 
sources  to  efficiently  make  decisions? 

•  Can  users  interact  with  ARES  without  the  use  of  peripheral  devices  (i.e., 
using  gestures)  and  what  benefit  does  this  type  of  interaction  provide? 

•  Do  some  users  perform  better  using  a  tangible  user  interface? 

•  Does  touching  or  shaping  the  sand  matter? 

•  What  do  the  findings  portend  for  other  systems  that  may  benefit  from  tactile 
feedback? 

•  How  does  user  and  team  performance  using  a  distributed  multiple-modality 
system  compare  with  a  distributed  single-modality  system? 

Generally,  research  questions  in  this  area  should  be  formulated  to  characterize  1) 
factors  moderating  the  experience  between  a  user  and  an  interface,  2)  the  quality  of 


Approved  for  public  release;  distribution  is  unlimited. 

8 


analysis  or  overall  performance  of  a  user  as  a  function  of  a  chosen  modality,  and/or 
3)  individual  and/or  team  performance  given  a  system  with  multiple  modalities 
and/or  multiple  users. 

2.4  Human  Performance  Assessment  Research  Area 

The  predominant  research  question  for  ARES  specifies  that  tools  developed  by  the 
platform  to  enhance  decision-making  attempt  to  best  meet  its  users’  requirements. 
Thus,  there  exists  the  need  to  accurately  model  human  performance  for  any  new 
modalities  or  information  visualization  techniques  employed  by  the  platform. 
Correctly  understanding  human  performance  can  assist  in  explaining  human 
variability  (Szalma  2009)  and  sources  of  human  error,  and  provide  predictions  for 
task  outcomes  and  the  behaviors  preceding  them.  Successful  performance  on  a  task 
requires  a  certain  degree  of  precision.  Humans  are  susceptible  to  internal  and 
external  factors  that  cause  them  to  be  imprecise.  Examples  of  these  factors  include 
aptitude,  existing  knowledge,  stress,  and  time  pressure  (DOE  2009). 

The  human  performance  assessment  research  area  facilitates  alignment  of  system 
parameters  and  capabilities  with  human  preferences  and  abilities  while  operating 
in  complex  environments.  This  allows  for  the  customization  of  interfaces  that  yield 
optimal  performance. 

The  literature  suggests  many  different  approaches  for  modeling  human 
performance.  The  effective  analysis  of  human  performance  requires  sufficient 
granularity  in  the  level  of  detail  associated  with  each  human  interaction. 
Quantitative  and  qualitative  methods  must  capture  this  detail  to  ensure  scientific 
validity.  Of  specific  emphasis  to  ARES  is  the  relationship  between  the  human  and 
the  system. 

Human  performance  assessment  tends  to  fall  under  3  specific  areas:  1)  perception 
and  attention  allocation,  2)  command  and  control,  and  3)  cognition  and  memory 
(Byrne  and  Pew  2009).  Within  attention  and  perception  are  human  factors 
fundamentals  such  as  signal  detection  theory  (i.e.,  does  a  stimulus  provide  enough 
information  to  distinguish  a  target  amongst  distractors?),  visual  search,  and 
multitasking  (Laughery  et  al.  2012).  Command  and  control  relies  on  the  ability  to 
select  information  in  an  efficient  and  effective  manner  (e.g.,  the  Hick-Hyman  law 
for  choice  response  time  [Hick  1952;  Hyman  1953]  and  the  observe,  orient,  decide, 
and  act  [OODA]  loop  to  model  associated  actions  [Gooley  and  McKneely  2012]). 
Cognition  and  memory  consists  of  the  understanding  of  acquiring  skills  and 
expertise  (i.e.,  cognitive  skill  acquisition),  the  interpretation  and  aggregation  of 
presented  information  (i.e.,  situation  awareness),  and  decision-making  as  it  is 
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related  to  ARES  users  (e.g.,  military  decision-making  process)  (Lickteig  et  al. 
1998). 


Potential  human  performance  assessment  research  questions  relevant  to  the  ARES 
program  include  the  following: 

•  Can  augmented  reality  capabilities  increase  performance  on  tasks  related  to 
battlespace  visualization? 

•  To  what  extent  does  augmented  reality  increase  performance  metrics  (e.g., 
accuracy,  time  on  task,  situational  awareness)  and  what  types  of  tasks  or 
scenarios  most  benefit  from  its  use? 

•  Can  the  ARES  platform  effectively  instruct  students  through  one-to-one  or 
one-to-many  methods  (e.g.,  virtual  avatars,  intelligent  tutors,  video 
teleconferences )  ? 

•  Do  certain  users’  spatial  abilities  make  a  difference? 

•  Are  there  differences  in  the  amount  of  information  users  retain  across 
various  modalities  (e.g.,  PowerPoint,  topographical  map,  traditional  sand 
table,  various  ARES  modalities)? 

•  What  are  the  generalizable  performance  predictors  across  tasks  as  users  are 
interacting  with  ARES? 

•  How  do  individual  difference  factors  such  as  self-efficacy,  motivation,  and 
personality  mediate  or  moderate  performance  outcomes? 

•  How  do  the  metrics  associated  with  individual  human  performance 
assessment  transfer  to  collaborative  learning  or  team  performance 
assessment  associated  with  ARES  interaction? 

•  What  standardized  human  performance  metrics  can  individual  researchers 
using  ARES  technologies  measure  to  ensure  the  ability  to  compare  across 
experiments? 

•  How  is  human  error  represented  within  ARES  technology  interaction  (i.e., 
active  errors  compared  to  latent  errors  and  slips  compared  to  mistakes)? 

Research  efforts  in  human  performance  assessment  will  also  need  to  focus  on 
addressing  both  the  strengths  and  weaknesses  of  assessment.  Strengths  include 
specificity,  clarity,  and  objectivity,  while  weaknesses  include  generalizability, 
validity,  and  confounding  variables  (Byrne  and  Pew  2009;  Creswell  2014).  The 
management  of  these  potential  dimensions  will  rely  on  a  combination  of  designing 
to  Soldier  requirements  while  ensuring  soundness  of  empirical  rigor. 
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2.5  Strategy  for  Establishing  the  Platform  and  Achieving  Basic 
Research  Goals 


The  3  research  areas  all  share  a  common  thread:  how  best  to  enable  the  human  user 
to  view  and  interact  with  content/information  so  that  they  can  better  understand  the 
information;  communicate  with  peers,  supervisors,  and  subordinates;  and  make 
better  and  faster  decisions.  A  challenge  in  crafting  any  research  strategy  for  a  new 
product  or  platform  is  appropriately  scoping  the  research,  demonstrating  the  value 
of  the  platform  while  simultaneously  performing  basic  and  applied  research  that  is 
relevant  beyond  the  platform.  To  resolve  pressing  immediate  questions,  an  effective 
strategy  might  first  ask,  “Is  this  worth  doing?”,  “Do  users  like  and  respond  better 
to  this  interface  or  technique?”,  and  “Is  performance  better  than  what  already 
exists?”  Once  the  value  of  the  platform  is  established,  follow-up  research  should 
ask  more  focused,  basic  research  questions  of  interest  to  the  greater  scientific 
community  that  also  yield  insight  to  improve  the  platform  and,  in  our  case,  provide 
value  to  the  Army. 


Figure  2  summarizes  the  overall  research  strategy,  showing  the  modalities,  research 
areas,  and  research  outcomes.  The  goal  of  the  ARES  research  program  is  to  balance 
the  practical  need  for  demonstrating  the  value  of  the  platform  and  its  enhancements 
while  also  conducting  basic  research  and  continuously  expanding  the  boundary  of 
the  applications  and  technologies  considered  within  the  platform.  Ultimately,  the 
ARES  platform  should  keep  the  enhancement  of  Army  readiness  as  its  ultimate 
goal. 


ARES 


Research  Areas 

Information 

Visualization 

Multi-Modal 

Interaction 

Human  Performance 
Assessment 


Outcomes 


Assessment  of 
Effectiveness 

Basic  and 
Applied 
Research 

Readiness 


Fig.  2  ARES  research  strategy 
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3.  Completed,  Ongoing,  and  Planned  ARES  Research 


Several  research  efforts  for  the  ARES  platform  have  already  been  completed  and 
published,  are  ongoing,  or  are  planned  as  of  the  end  of  fiscal  year  (FY)  2017  (the 
time  of  writing  for  this  technical  note).  Table  1  lists  these  efforts  and  breaks  them 
down  by  research  area  and  time  frame.  The  subsequent  paragraphs  describe  the 
principal  investigator(s),  start/completion  dates,  and  an  abstract/executive 
summary  for  each  effort.  Note  that  actively  planned  work  is  included  in  the 
“Planned”  category,  but  not  conceptual  future  work.  Generally,  the  criteria  used  to 
determine  the  planned  research  included  here  are  the  following:  1)  the  purpose  of 
the  study  is  known  and  some  objectives  have  been  established  that  relate  to  one  or 
more  ARES  research  questions,  2)  the  means  of  achieving  the  objectives  have  been 
considered,  and  3)  the  researchers  who  will  perform  the  work  have  been  identified 
(or  a  contract  is  in  place  that  could  support  the  research).  Also  note  that  almost  all 
of  the  research  crosses  all  3  research  areas,  but  the  table  only  indicates  each  study’s 
primary  area  of  inquiry. 

Table  1  Completed,  ongoing,  and  planned  (short-term)  ARES  studies,  organized  by 
research  area 


Research 

areas 

Completed 

Ongoing 

Planned 

Information 

Visualization 

N/A 

4.  Chem-Bio  Model 
Visualization 

7.  Terrain  Correlation 
Guidance 

Multi-Modal 

Interaction 

N/A 

5.  Cognitive  Impact 
of  Tangible  AR 

8.  Floor  Projection 

Human 

Performance 

Assessment 

1 .  Impact  on  Learning 

2.  Tactics  (Pilot) 

3.  Tactics  II 

6.  Time  and 

Accuracy 

9.  Impact  on  Learning  II: 
Expansion  to  AR/VR 

10.  Land  Navigation 

11.  Distributed 
Collaboration 

3.1  Completed  Studies 

3.1.1  Study  1:  Impact  on  Learning 

Primary  Research  Area:  Human  Performance  Assessment 

Principal  Investigator:  Dr  Kelly  Hale  (Design  Interactive,  Inc.) 
Completion  Date:  Q1  FY16 
Abstract/Executive  Summary: 
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The  goal  of  this  study  was  to  empirically  examine  spatial  knowledge  acquisition 
and  understanding  using  traditional  media  (2-D  paper,  topographic  map),  a  3-D 
terrain  rendering  on  a  PC,  and  the  Augmented  REality  Sandtable  (ARES).  The  main 
objective  of  the  study  was  to  determine  the  impact  of  the  various  media  on  spatial 
knowledge  acquisition  and  spatial  reasoning  skills.  Results  from  the  study  indicate 
that  ARES  supported  improved  landmark  identification  and  distance  estimation  as 
compared  to  the  traditional  alternatives;  users  also  provided  high  ratings  of 
perceived  utility  for  the  ARES  platform.  The  study  participants  included  43 
students  from  the  University  of  Central  Florida’s  (UCF)  Reserve  Officers’  Training 
Corps  (ROTC)  and  7  Reserve  personnel  from  the  US  Army’s  143rd  Sustainment 
Command  in  Orlando,  Florida. 

Publications'. 

•  Schmidt-Daly  TN,  Riley  JM,  Hale  KS,  Yacht  D,  Hart  J.  Augmented  REality 
Sandtable’s  (ARES)  impact  on  learning.  Adelphi  (MD):  Army  Research 
Laboratory  (US);  2016.  Report  No.:  ARL-CR-0803. 

3.1.2  Study  2:  Tactics  (Pilot) 

Primary  Research  Area:  Human  Performance  Assessment 
Principal  Investigator:  Dr  Michael  Boyce  (ARL) 

Completion  Date:  Q2  FY16 
Abstract/Executive  Summary: 

This  research  project  integrated  the  Generalized  Intelligent  Framework  for 
Tutoring  (GIFT)  and  the  Augmented  REality  Sandtable  (ARES)  for  the  assessment 
of  military  tactics.  An  experiment  involving  19  students  from  the  University  of 
Central  Florida’s  (UCF)  Reserve  Officers’  Training  Corps  (ROTC)  assessed 
performance,  physiological,  and  experiential  data  in  a  between-subjects  design.  The 
conditions  consisted  of  a  2-D  map  displayed  on  either  a  flat  or  contoured  surface 
(sand  table),  both  of  which  leveraged  the  projection  technology  of  ARES.  Results 
of  the  study  did  not  indicate  significant  differences  between  time  on  task,  accuracy, 
or  electrodermal  activity,  but  a  larger  sample  size  is  needed  to  verify  findings. 
Preference  between  conditions  was  more  prevalent  in  support  of  ARES;  however, 
individuals  that  preferred  the  flat  condition  discussed  issues  with  the  ARES 
condition  being  more  difficult  to  precisely  measure,  consistent  with  the  findings  in 
the  literature. 

Publications: 
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•  Boyce  MW,  Reyes  RJ,  Cruz  DE,  Ambum  CR,  Goldberg  B,  Moss  JD, 
Sottilare  RA.  Effect  of  topography  on  learning  military  tactics  -  integration 
of  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  and  Augmented 
REality  Sandtable  (ARES).  Aberdeen  Proving  Ground  (MD):  Army 
Research  Laboratory  (US);  2016.  Report  No.:  ARL-TR-7792. 

•  Boyce  MW,  Goldberg  B,  Moss  JD.  Electrodermal  activity  analysis  for 
training  of  military  tactics.  Proceedings  of  the  Human  Factors  and 
Ergonomics  Society  Annual  Meeting;  2016;  Sage,  CA.  Los  Angeles  (CA): 
SAGE  Publications,  vol.  60;  No.  1;  p.  1339-1343. 

•  Boyce  MW.  From  concept  to  publication  -  a  successful  application  of  using 
GIFT  from  the  ground  up.  Generalized  Intelligent  Framework  for  Tutoring 
(GIFT)  Users  Symposium  (GIFTSym4);  2016  Jan.  p.  125. 

•  Boyce  MW,  Cruz  D,  Sottilare  R.  Interpretative  phenomenological  analysis 
for  military  tactics  instruction.  In:  Kantola  J,  Barath  T,  Nazir  S,  Andre  T, 
editors.  Advances  in  human  factors,  business  management,  training  and 
education.  Springer  International  Publishing;  2017.  vol.  498;  p.  623-634. 

3.1.3  Study  3:  Tactics  II 

Primary  Research  Area:  Human  Performance  Assessment 
Principal  Investigator:  Dr  Michael  Boyce  (ARL) 

Completion  Date:  Q2  FY17 
Abstract/Executive  Summary: 

This  was  a  follow-on  effort  to  Study  2.  Experimentation  assessed  how  displaying 
information  onto  different  surfaces  can  influence  the  performance  and  engagement 
of  cadets  in  answering  questions  on  military  tactics.  The  study  used  2  experimental 
conditions:  flat  2-D  projection  vs.  the  Augmented  REality  Sandtable  (ARES). 
Objective  performance  measures  included  time  on  task  and  accuracy  while  self- 
report  surveys  and  electrodermal  activity  measured  engagement.  There  were  62 
cadets  recruited  through  the  United  States  Military  Academy  (USMA).  Data 
collection  is  complete  and  analysis  is  underway  as  of  Q4  FY 17. 

Publications'. 

•  Boyce  MW,  Rowan  CP,  Moss  JD,  Amburn  CR,  Shorter  PL,  Garneau  CJ, 
Sottilare  RA.  The  impact  of  surface  projection  on  military  tactics 
comprehension.  Military  Psychology.  Forthcoming  2017. 
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3.2  Ongoing  Studies 


3.2.1  Study  4:  Chem-Bio  Model  Visualization 

Primary  Research  Area:  Information  Visualization 

Principal  Investigator:  Dr  Jennifer  Murphy  (Quantum  Improvements  Consulting) 
Start  Date:  Q2  FY 1 8 
Abstract/Executive  Summary: 

This  Defense  Threat  Reduction  Agency  (DTRA)-funded  study  uses  new 
Augmented  REality  Sandtable  (ARES)  visualizations  of  chemical  and  biological 
agent  propagations  across  user  shaped  terrains  on  the  physical  sand  table.  By 
employing  these  new  visualizations  for  use  in  the  Army’s  Chemical,  Biological, 
Radiological,  and  Nuclear  School’s  Captains  Career  Course,  the  purpose  of  this 
study  is  to  better  understand  how  interaction  with  ARES  may  result  in  deeper 
representations  and  understanding  of  terrain  and  weather  effects  on  plume 
propagations.  It  is  hypothesized  that  interacting  with  the  new  visualizations  on 
ARES  will  enable  students  to  produce  better-quality  deliverables  in  less  time,  with 
higher  scores  on  knowledge  assessment,  and  better  information  retention  over  time. 
Additional  hypotheses  will  assess  cognitive  demand,  attitude,  and  technology 
acceptance  for  students  using  the  ARES  visualization  and  traditional  learning 
materials. 

3.2.2  Study  5:  Cognitive  Impact  of  Tangible  AR 

Primary  Research  Area:  Multi-Modal  Interaction 

Principal  Investigators:  Dr  Michael  Boyce  (ARL),  Dr  Aaron  Gardony  (NSRDEC) 
Start  Date:  Q4  FY  17 
Abstract/Executive  Summary: 

This  study  examines  how  the  physical  ability  to  touch  and  manipulate  sand 
influences  cognitive  processes  that  critically  underlie  mission  planning  and  mission 
success,  as  compared  to  passively  observing  the  same  action.  It  employs  a 
systematic  empirical  investigation,  emphasizing  quantitative  metrics.  Specifically, 
the  influence  of  tangibility  afforded  by  the  Augmented  REality  Sandtable  (ARES) 
on  the  development  of  spatial  mental  representations  of  complex  3-D  terrain  is 
investigated.  Data  collection  trials  in  this  study  entail  96  participants  grouped  in 
dyads  and  then  instructed  to  either  build  terrain  features  on  the  ARES  or  to  observe 
another  participant  as  they  build  the  terrain  features.  Assessment  on  their 
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knowledge  of  the  terrain,  as  well  as  their  ability  to  recall  the  terrain  they  built  in  the 
presence  of  distractors,  will  take  place  after  performing  their  respective  tasks.  The 
primary  independent  variable  is  whether  the  participant  is  an  observer  or  a  builder, 
while  the  dependent  variables  consist  of  terrain  knowledge  (both  conceptual  and 
verification  of  terrain)  as  well  as  spatial  memory  performance. 

3.2.3  Study  6:  Time  and  Accuracy 

Primary  Research  Area:  Human  Performance  Assessment 

Principal  Investigator:  Dr  Kelly  Hale  (Design  Interactive,  Inc.) 

Start  Date:  Q4  FY 17 
Abstract/Executive  Summary: 

The  objective  of  this  study  is  to  evaluate  the  effectiveness  and  efficiency  of  sand 
table  terrain  model  construction  using  the  Augmented  REality  Sandtable  (ARES) 
compared  to  traditional  sand  table  methods,  and  to  gather  perceptions  and  objective 
data  on  the  usability  of  the  ARES  software,  technology,  and  interface  design.  The 
study  participants  were  Soldiers  from  the  Army’s  3rd  Infantry  Division.  The 
apparatus  used  was  the  7-  x  4-ft  physical  sand  table,  ARES  software,  and  associated 
mobile  tablet  for  the  experimental  condition  (using  ARES),  whereas  the  control 
condition  (traditional  sand  table)  used  a  replica  of  the  7-  x  4-ft  sand  box  and  an 
associated  terrain  building  kit.  Materials  included  an  informed  consent,  a 
demographics  questionnaire,  a  written  land  navigation  test  to  evaluate  construction 
aptitude,  data  recording  forms  to  capture  sand  table  construction  performance,  and 
questionnaires  that  measure  perceived  usability  and  capture  feedback  on  tool 
utility. 

3.3  Planned  Studies 


3.3.1  Study  7:  Terrain  Correlation  Guidance 

Primary  Research  Area:  Information  Visualization 

Principal  Investigator:  Dr  Stephen  Serge  (University  of  Central  Florida  Institute 
for  Simulation  &  Training) 

Anticipated  Start  Date:  Q1  FY18 

Abstract/Executive  Summary: 

User  interaction  with  the  Augmented  REality  Sandtable  (ARES)  involves  direct 
user  manipulation  of  the  sand  to  conform  to  specific  terrain  features  projected  onto 
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the  sand.  Currently,  the  table  is  capable  of  providing  a  “rough  guide”  to  users  in 
order  to  adjust  the  elevation  of  the  sand.  This  helps  users  set  up  a  table  that  reflects 
the  terrain  features  of  the  particular  map  area  in  order  to  provide  a  deeper  level  of 
realism  and  accuracy.  One  of  the  main  goals  of  creating  this  representation  of  the 
Area  of  Operation  (AO)  is  to  instill  in  the  users  a  sense  of  familiarity  with  the  AO 
prior  to  running  any  operations.  The  initial  push  of  this  research  effort  focuses  on 
quickly  compiling  and  developing  a  research- supported  approach  toward  the 
development  of  a  new  integrated  guided  terrain  build  feature  within  the  ARES 
software.  Depending  on  outcomes,  additional  research  goals  may  be  added  in  a 
second  phase  of  research. 

3.3.2  Study  8:  Floor  Projection 

Primary  Research  Area:  Multi-Modal  Interaction 

Principal  Investigator:  Dr  Gregory  Welch  (University  of  Central  Florida  Institute 
for  Simulation  &  Training) 

Anticipated  Start  Date:  Q1  FY18 

Abstract/Executive  Summary: 

The  goal  of  the  proposed  work  is  for  the  development  of  a  large-area  floor 
projection  to  extend  the  visible  range  and  interaction  capabilities  of  the  Augmented 
REality  Sandtable  (ARES).  Several  research  questions  will  be  addressed  in  the 
formulation  of  the  study  plan  as  follows:  1)  What  are  the  appropriate  human  factors 
tools  and  techniques  (as  measured  by  cost  and  ease  of  use)  to  create  intuitive  and 
usable  interaction  with  large-scale  projection?  2)  What  is  the  impact  on  cognition 
(i.e.,  load,  processing,  and  awareness)  of  large-scale  projection  as  compared  to 
smaller  form  factor  interfaces?  3)  How  can  information  be  displayed  to  facilitate 
distributed  collaboration  across  geographically  dispersed  teams  as  well  as  to  large 
groups  of  individuals  viewing  the  display?  4)  When  presented  with  multiple  ways 
to  view  information  across  platforms  and  form  factors,  what  is  the  process  for 
interaction  as  a  user  is  switching  between  views?  5)  How  can  the  technology 
infrastructure  be  used  to  increase  the  scalability  and  generalizability  to  support 
variable  system  configurations  based  on  environmental  constraints? 

3.3.3  Study  9:  Impact  on  Learning  II:  Expansion  to  AR/VR 

Primary  Research  Area:  Human  Performance  Assessment 

Principcd  Investigator:  Dr  Kelly  Hale  (Design  Interactive,  Inc.) 

Anticipated  Start  Date:  Q1  FY18 
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Abstract/Executive  Summary: 


The  goal  of  the  proposed  work  is  to  conduct  an  evaluation  of  the  Augmented 
REality  Sandtable  (ARES)  (as  viewed  through  an  AR  peripheral  such  as  the 
Microsoft  Hololens)  and  compare  results  to  data  collected  in  Study  1.  The  proposed 
study  will  be  run  as  a  between-subjects  investigation  to  evaluate  differences 
between  multimedia  displays  of  sand  tables  on  spatial  knowledge  and  decision¬ 
making  tasks. 

3.3.4  Study  10:  Land  Navigation 

Primary  Research  Area:  Human  Performance  Assessment 

Principal  Investigator:  Dr  Ben  Goldberg  (ARL) 

Anticipated  Start  Date:  Q2  FY18 
Abstract/Executive  Summary: 

The  purpose  of  this  study  is  to  use  the  Augmented  REality  Sandtable  (ARES)  as  a 
use  case  with  the  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  for  a  land 
navigation  task.  Participants  will  be  asked  to  plan  a  route  and  place  map  symbology 
in  the  correct  locations  on  the  map.  Participants  will  be  cadets  from  the  United 
States  Marine  Academy.  Real-time  user  performance  assessments  and  feedback 
from  GIFT  will  be  provided  to  participants.  Study  metrics  will  include  objective 
and  subjective  measures  of  learning  and  performance  (e.g.,  workload,  engagement, 
usability)  while  interacting  with  the  various  ARES  modalities  (and  also  perhaps 
traditional/2-D  alternatives). 

3.3.5  Study  11:  Distributed  Collaboration 

Primary  Research  Area:  Human  Performance  Assessment 

Principal  Investigator.  Dr  Gregory  Welch  (Tentative;  University  of  Central  Florida 
Institute  for  Simulation  &  Training) 

Anticipated  Start  Date:  Q3  FY18 

Abstract/Executive  Summary: 

The  purpose  of  this  study  is  to  research  methods  to  improve  distributed 
collaboration  scenarios  with  multiple  users  interacting  with  the  Augmented  REality 
Sandtable  (ARES)  systems  in  geographically  distributed  locations.  With  new 
mixed  reality  tools  being  implemented  and  evaluated  across  the  Department  of 
Defense,  we  want  to  know  how  well  users  understand,  discuss,  and  collaborate  on 
complex  battlespaces  (e.g.,  kinetic,  cyber,  and  social)  using  mixed  reality.  Two 
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testbeds,  already  in  place  at  2  remote  locations  for  use  by  Reserve  Officers’ 
Training  Corps  (ROTC)  students,  will  use  a  server  to  share  scenario  visualization 
and  edit  with  video  teleconference  capabilities  to  support  communications.  Human 
performance  metrics  (e.g.,  workload,  engagement,  usability)  will  be  evaluated, 
along  with  measures  of  effectiveness  of  communication  and  post-knowledge  of  the 
resultant  plan. 

3.4  Summary 

Together,  these  1 1  completed,  ongoing,  or  planned  studies  represent  efforts  that 
seek  to  establish  the  effectiveness  of  the  ARES  platform,  expand  to  new 
applications  and  points  of  need,  and  explore  new  technologies  and  capabilities 
while  performing  research  that  answers  questions  of  greater  scientific  interest  in 
information  visualization,  multimodal  interaction,  and  human  performance 
assessment. 

4.  Conclusion 


The  ARES  platform  provides  an  opportunity  to  bring  together  multiple 
technologies  for  user  interaction — including  virtual  and  augmented  reality,  a 
tangible  interface,  and  other  distributed  modalities — to  improve  battlespace 
visualization  and  decision-making  to  provide  a  common  operating  picture  at  the 
point  of  need.  The  purpose  of  this  technical  note  is  to  situate  completed,  ongoing, 
planned,  and  future  work  into  the  overall  topology  of  research  for  the  ARES 
platform.  Future  work  in  the  various  research  areas  will  answer  questions  of  broad 
scientific  interest  and  will  establish  ARES  as  a  platform  for  exciting  developments. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


2-D 

2-dimensional 

3-D 

3-dimensional 

AO 

area  of  operations 

ARL 

US  Army  Research  Faboratory 

ARES 

Augmented  REality  Sandtable 

DTRA 

Defense  Threat  Reduction  Agency 

FY 

fiscal  year 

GIFT 

Generalized  Intelligent  Framework  for  Tutoring 

KCI 

Key  Campaign  Initiatives 

NSRDEC 

US  Army  Natick  Soldier  Research,  Development  and 
Engineering  Center 

OODA 

observe,  orient,  decide,  and  act 

PC 

personal  computer 

ROTC 

Reserve  Officers’  Training  Corps 

SWAT 

Special  Weapons  and  Tactics 

UCF 

University  of  Central  Florida 

UDOP 

User-Defined  Common  Operating  Picture 

USMA 

United  States  Military  Academy 
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